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The  Software  Supportability  Qualitative  Assessment  Methodology  is  a  five 
volume  reference  set  that  provides  measures  to  aid  in  the  support  of  information  systems. 
These  manuals  are  aimed  at  improving  the  support  process  by  more  accurately  assessing  the 
capabilities  of  support  organizations,  quantitatively  measuring  the  supportability  of  fielded 
systems  and  evaluating  the  operational  readiness  of  fielded  systems. 

Volume  I,  Developing  Quality  Measures  for  Information  Systems  Support ,  describes  the 
three  measures  along  with  the  model  of  information  system  support  that  the  measures  axe 
designed  to  satisfy.  This  is  the  main  volume  of  the  set  and  should  be  consulted  before 
implementing  the  measures  described  in  more  detail  in  the  other  volumes. 

Volume  II,  The  Review  of  Metrics  for  Developing  an  Information  Systems  Support  Mea¬ 
surement  Framework,  provides  a  survey  and  evaluation  of  current  metrics  in  terms  of  in¬ 
formation  systems  support.  Specifically,  three  classes  of  metrics  are  reviewed:  software 
product  metrics,  life  cycle  process  metrics,  and  process  management  metrics. 

Volume  III,  Implementing  the  Software  Supportability  Measure ,  provides  instructions  for 
collecting  data  for  the  measure,  compiling  the  measure  by  evaluating  the  data,  and  inter¬ 
preting  the  final  result.  The  volume  also  contains  guidelines  for  improving  the  supportabilty 
of  an  information  system  based  on  its  evaluation.  Specifically,  the  volume  contains  resource 
estimations  for  compiling  and  evaluating  the  measure,  questionnaires  for  collecting  the  re¬ 
quired  data  and  step-by-step  instructions  for  measuring  the  supportability  of  an  information 
system. 

Volume  IV,  Implementing  the  Support  Organization  Assessment  Measure,  provides  in¬ 
structions  for  collecting  data  for  the  assessment,  conducting  the  assessment,  and  interpret¬ 
ing  the  final  result.  The  volume  also  contains  guidelines  for  improving  the  capabilities  of 
a  support  organization  based  on  its  evaluation.  Specifically,  the  volume  contains  resource 
estimations  for  conducting  and  evaluating  the  assessment,  questionnaires  for  collecting  the 
required  data  and  6tep- by-step  instructions  for  measuring  the  capabilities  of  a  support  or¬ 
ganization. 

Volume  V,  Implementing  the  Operational  Readiness  Measure,  provides  instructions  for 
collecting  data  for  the  measure,  compiling  the  measure  by  evaluating  the  data,  and  inter¬ 
preting  the  final  result.  The  volume  also  contains  guidelines  for  improving  the  operational 
readiness  of  an  information  system  based  on  its  evaluation.  Specifically,  the  volume  contains 
resource  estimations  for  compiling  and  evaluating  the  measure,  questionnaires  for  collecting 
the  required  data  and  step-by-step  instructions  for  measuring  the  operational  readiness  of 
an  information  system. 
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1  Introduction 


The  software  supportability  measure  is  focused  on  defining  the  important  factors  that  affect  the 
suppoTtability  of  a  fielded  information  system.  Software  supportability  is  a  measure  of  the  effort 
required  to  satisfy  users  expectations  of  a  given  software  product,  where  user  expectations  can  be 
divided  into  two  groups.  First  the  users  expect  the  software  to  fulfill  its  intended  functions,  i.e. 
its  requirements.  Second,  users  generally  expect  the  software  to  meet  new  requirements.  Factors 
affecting  the  effort  required  to  satisfy  these  expectations  can  be  divided  into  three  categories: 
the  software  product  itself,  the  available  resources  for  support  activities,  and  the  management 
procedures  used  to  guide  the  support  process. 

The  purpose  of  this  measure  is  to  give  the  support  organization  a  rough  characterization  of  the 
supportability  of  an  information  system  supported  by  the  organization.  The  measure  is  made  up 
three  factors:  system,  process,  and  resource.  The  system  factor  measures  criteria  related  solely 
to  the  information  system.  The  process  factor  measures  components  related  to  the  maturity 
and  effectiveness  of  the  process  used  to  guide  system  support.  The  resource  factor  measures 
components  related  to  the  availability  and  effectiveness  of  resources  critical  to  system  support. 

This  document  describes  how  to  measure  the  supportability  of  an  information  system  that  is 
maintained  by  your  organization.  The  second  section  details  what  resources  in  time,  material, 
and  personnel  are  required  to  compute  this  measure.  The  following  sections  describe  the  process 
for  computing  and  interpreting  the  supportability  measure. 

It  is  important  you  read  through  the  following  two  sections  (up  through  Calculating  and 
Evaluating  the  Supportability  Measure)  before  beginning  this  process.  It  is  also  important  that 
any  personnel  who  will  provide  data  for  this  process  (by  completing  one  or  more  questionnaires) 
do  NOT  read  the  sections  on  scoring  the  questionnaires  and  evaluating  the  results  (Sections  4 
and  5)  until  after  they  have  completed  the  questionnaires. 


2  Requirements 

Material 

Little  in  the  way  of  materials  is  required  to  conduct  this  exercise.  Appendices  C  and  D  in  this 
volume  contain  the  Organization  and  System  questionnaire.  These  should  be  photocopied  and 
distributed  to  the  appropriate  personnel  (see  next  subsection).  Appendix  E  contains  directions 
and  worksheets  for  scoring  the  questionnaires.  A  set  of  directions  and  a  worksheet  should  be 
photocopied  for  each  questionnaire.  Appendix  F  contains  a  worksheet  that  should  be  used  to 
record  the  final  results  of  the  supportability  measurement.  This  worksheet  should  be  photocopied 
and  should  be  used  in  interpreting  and  evaluating  the  final  measurement. 


Audience 

The  careful  selection  of  the  appropriate  personnel  to  complete  the  organization  and  system  ques¬ 
tionnaires  is  critical  to  the  success  of  the  measure.  An  examination  of  the  questionnaires  should 
be  the  first  step  in  choosing  your  audience.  In  general,  the  organization  questionnaire  should  be 
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completed  by  personnel  tasked  with  managing  the  support  process.  Likewise  the  system  ques¬ 
tionnaire  should  be  completed  by  personnel  tasked  with  the  actual  maintenance  of  the  system. 

Recognizing  that  a  significant  portion  of  the  questionnaires  ask  for  subjective  answers,  dis¬ 
tributing  the  questionnaires  to  a  set  of  respondents  and  averaging  their  scores  should  reduce  bias 
accompanying  subjective  responses. 

Selecting  a  coordinator  to  distribute,  collect,  validate,  and  score  the  questionnaires  is  required. 
The  coordinator  is  responsible  for  distributing  the  questionnaires  and  answering  any  remaining 
questions  the  respondents  may  have.  The  coordinator  must  also  collect  the  questionnaires,  veri¬ 
fying  that  all  questions  have  been  answered  completely.  The  coordinator  must  also  validate  the 
questionnaires  against  each  other.  Essentially  the  coordinator  assures  that  the  answers  make 
sense  (i.e.  percentages  add  up  to  100)  and  that  the  respondents  interpreted  the  questions  in  the 
same  manner.  More  information  on  this  process  can  be  found  in  the  next  section.  Finally  the 
coordinator  is  the  best  person  to  be  responsible  for  scoring  the  questionnaire  and  compiling  the 
final  results.  The  coordinator  may  NOT  complete  any  of  the  questionnaires  as  a  respondent. 

In  summary,  this  process  requires  a  minimum  of  three  personnel  to  answer  the  organization 
and  system  questionnaire  and  to  serve  as  coordinator,  respectively.  The  final  results  will  be  more 
meaningful  if  two  sets  of  people  (possibly  overlapping)  complete  the  two  questionnaires. 

Time 

The  amount  of  time  required  to  conduct  the  measurement  depends  on  two  factors:  the  amount  of 
on-line  or  easily  accessible  data  and  the  number  of  personnel  tasked  to  complete  questionnaires. 

The  organizational  questionnaire  should  take  from  4  person-hours  to  24  person-hours  to  com¬ 
plete.  The  size  of  the  organization  and  the  amount  of  easily  accessible  information  about  the 
staff  determines  this  variation. 

The  system  questionnaire  should  take  from  4  person-hours  to  12  person-hours  to  complete. 
Again,  the  amount  of  readily  accessible  information  determines  this  variation. 

Each  questionnaire  should  take  one  half  of  a  person-hour  to  score. 

The  amount  of  time  required  of  the  coordinator  is  determined  by  the  number  of  personnel 
filling  out  questionnaires.  Validating  the  questionnaires  should  take  approximately  one  person- 
hour  per  questionnaire.  The  effort  should  be  less  for  a  small  number  of  questionnaires. 

A  rough  formula  to  calculate  the  time  required  for  collecting  the  data  and  calculating  the 
measure  is  given  on  the  next  page. 
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OW  ■  Weight  based  on  size  of  organization  and  accessibility  of  information 
for  the  staff.  Range  from  1  (small  organization  and  readily  accessible 
information)  to  6  (large  organization  etc..  ). 

SW  ■  Weight  based  accessibility  of  information  for  the  system. 

Range  from  1  (readily  accessible  information)  to  3. 

ON  •  Number  of  organizational  questionnaires  completed. 

SN  •  Number  of  systems  questionnaires  completed. 


TT  >  Total  time  required  in  person-hours. 


TT  •  4*0W*0N  +  4*SW*SN  ♦  (ON  *  SN)  *  .6* (ON  +  SN) 

I  I  I  I 

I  I  I  scoring 

I  I  I 

I  I  -  validation 

I  I 

I  -  completing  system  questionnaires 

I 

-  completing  organizational  questionnaires 


3  Calculating  and  Evaluating  the  Supportability  Measure 

This  process  consists  of  six  steps. 

1.  Select  respondents  and  coordinator. 

2.  Review  questionnaires  (optional). 

3.  Fill  out  questionnaires. 

4.  Validate  questionnaires. 

5.  Score  questionnaires,  compute  measure. 

6.  Interpret  final  result. 

First,  the  personnel  tasked  with  completing  the  questionnaires  and  the  overall  coordinator 
should  be  selected.  Refer  to  the  earlier  section  for  guidelines  for  selecting  questionnaire  respon¬ 
dents.  It  is  possible,  even  desirable,  for  the  two  sets  of  respondents  to  overlap.  In  other  words, 
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some  people  could  fill  out  both  system  and  organization  questionnaires.  This  overlap  enables  dif¬ 
ferent  perspectives  to  be  incorporated.  A  portion  of  the  organizational  questionnaire  (questions 
5-8)  concerns  educational  and  other  historical  information  about  the  application  staff.  This 
information  may  be  inaccessible  to  some  respondents.  If  necessary,  this  information  could  be 
supplied  by  other  person,  perhaps  the  coordinator,  since  this  information  is  strictly  quantitative 
and  not  subjective. 

The  system  questionnaire  included  in  this  volume  is  the  same  questionnaire  that  is  used 
to  compile  the  operational  readiness  measure  for  an  information  system  (see  Volume  V).  To 
compute  the  supportability  measure  only  the  starred  (+)  questions  (questions  1- 
10, 14, 15,10,20a- j,20o)  need  to  be  completed.  If  you  are  computing  both  the  supportability 
and  operational  readiness  measures,  the  entire  system  questionnaire  needs  to  be  completed. 

Optionally,  after  the  respondents  have  been  selected,  a  meeting  could  be  held  to  review  the 
questionnaires.  This  meeting  should  be  led  by  the  coordinator.  The  purpose  of  this  meeting 
is  to  assure  the  respondents  understand  the  questions  in  the  same  manner.  Discussions  about 
possible  answers  should  not  be  permitted.  Only  definitional  information  should  be  distributed. 
The  advantage  of  this  meeting  is  that  it  should  quicken  the  completion  of  the  questionnaires  by 
the  respondents  and  it  should  reduce  the  variability  in  their  interpretation  of  the  questions. 

Next,  the  questionnaires  should  be  completed.  The  coordinator  should  be  available  to  answer 
questions  of  interpretation.  Respondents  should  be  encouraged  to  write  comments  concerning 
interpretation  next  to  their  answers.  This  effort  will  aid  the  coordinator  in  validating  the  ques¬ 
tionnaires. 

The  questionnaires  should  be  returned  to  the  coordinator  who  should  attempt  to  validate  the 
responses.  First,  all  questions  should  be  answered!  Second,  responses  containing  quantitative, 
non-subjective  data  should  correspond  closely  if  not  be  equivalent.  Third,  the  coordinator  should 
look  for  differing  interpretations  by  examining  comments  added  by  the  respondents. 

Next,  the  coordinator  should  score  the  questionnaires.  Refer  to  the  next  section  and  ap¬ 
pendices  E  and  F  for  scoring  directions.  The  system  questionnaire  scoring  directions  contains 
only  directions  for  scoring  the  questions  needed  for  the  supportability  measure.  These  directions 
should  be  used  for  computing  the  supportability  measure.  Scoring  directions  for  computing  the 
operational  readiness  measure  can  be  found  in  Volume  V. 

The  coordinator  should  average  the  scores  to  compute  a  final  score.  Finally,  the  coordinator 
and  other  personnel  should  consult  Section  5  in  order  to  interpret  the  final  result. 


4  Scoring  the  Questionnaire  Answers 

Appendices  E  and  F  contain  directions  for  scoring  the  individual  questionnaires  and  computing 
the  final  measure.  Essentially,  each  answer  will  correspond  to  a  certain  number  of  points.  Scoring 
directions  for  each  question  are  provided  with  possible  ranges  specified.  The  results  for  each  ques¬ 
tionnaire  are  then  recorded  on  the  questionnaire  worksheets.  The  worksheets  divide  the  responses 
among  three  categories:  System,  Process,  and  Resource.  These  categories  are  the  three  major 
factors  of  the  supportability  measure.  The  columns  for  each  category  should  be  totaled.  The 
totals  for  the  organizational  questionnaires  should  then  be  averaged  by  the  number  of  organiza¬ 
tional  questionnaires  completed;  likewise  for  the  system  questionnaires.  These  averages  should  be 
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recorded  os  the  Supportability  Worksheet.  Instructions  on  the  Supportability  Worksheet  should 
then  be  followed  to  compute  the  final  result. 


5  Interpreting  the  Results 


Software  supportability  is  a  measure  of  the  effort  required  to  satisfy  users  expectations  of  a  given 
software  product,  where  user  expectations  can  be  divided  into  two  groups.  First  the  users  expect 
the  software  to  fulfill  its  intended  functions,  i.e.  its  requirements.  Second,  users  generally  expect 
the  software  to  mee;  new  requirements.  Factors  affecting  the  effort  required  to  satisfy  these  ex¬ 
pectations  can  be  divided  into  three  categories:  the  software  product  itself,  the  available  resources 
for  support  activities,  and  the  management  procedures  used  to  guide  the  support  process. 


The  Supportability  Measure 

The  supportability  measure  for  an  information  system  can  be  interpreted  in  two  ways.  First, 
it  can  be  interpreted  as  the  percentage  chance  of  successfully  fulfilling  user  expectations.  This 
interpretation  is  rather  severe,  though.  The  following  ranges  may  be  more  meaningful: 

76-100  Excellent  rating,  low  risk  system 
61-76  Good  rating,  average  risk  system 
26-60  Mediocre  rating,  requires  attention  to  reduce  risk 
7-26  Poor  rating,  requires  immediate  attention 

The  final  measurements  for  each  factor  of  the  supportability  measure  are  also  meaningful. 
The  system  factor  measures  components  related  solely  to  the  information  system.  The  process 
factor  measures  components  related  to  the  maturity  and  effectiveness  of  the  process  used  to 
guide  system  support.  The  resource  factor  measures  components  related  to  the  availability  and 
effectiveness  of  resources  critical  to  system  support. 

The  System  Factor 

Criteria  for  the  system  factor  considers  the  existence  and  quality  of  system  documentation,  the 
age  and  change  history  of  the  system,  the  complexity,  size  and  modularity  of  the  source  code, 
the  existence  of  debugging  code  and  adequate  testing,  the  urgency  of  change  requests,  and  the 
quality  of  the  original  system.  The  final  score  for  the  system  factor  can  range  from  2  to  100.  The 
following  interpretation  may  be  useful: 

76-100  Excellent  rating,  low  risk  system 
61-75  Good  rating,  average  risk  system 
26-50  Mediocre  rating,  implement  improvements 
2-25  Poor  rating,  system  not  supportable 
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Depending  on  the  severity  of  the  rating,  the  following  actions  may  be  taken  to  improve  the 
system  factor.  This  list  does  not  represent  all  possible  appropriate  actions. 

•  Re-design  the  system  (reverse  engineering). 

•  Create/update  system  documentation. 

•  Increase  testing  time. 

•  Accept  fewer  emergency  change  packages. 

•  Consistently  use  regression  testing. 

The  Process  Factor 

Criteria  for  the  process  factor  considers  the  existence  and  and  adequacy  of  effective  organizational 
techniques,  the  existence  and  adequacy  of  important  standards,  the  existence  of  useful  work 
methods,  training  of  the  user  population,  and  adequate  forecasting  of  resource  requirements. 
The  final  score  for  the  process  factor  can  range  from  0  to  100.  The  following  interpretation  may 
be  useful: 

76-100  Excellent  rating;  mature,  effective  process 
61-75  Good  rating,  average  process 
28-50  Mediocre  rating,  implement  improvements 
0-25  Poor  rating,  process  requires  immediate  attention 

Process  improvements  are  difficult  to  make.  Examining  questionnaire  responses  may  provide 
clearer  direction.  Depending  on  the  severity  of  the  rating,  the  following  actions  may  be  taken  to 
improve  the  process  factor.  This  list  does  not  represent  all  possible  appropriate  actions. 

•  Incorporate  known  effective  organizational  techniques  such  as  maintenance  escort,  accep¬ 
tance  review,  or  object-oriented  design. 

•  Create  /  update  needed  standards. 

•  More  strictly  enforce  standards. 

•  Increase  user  training. 

•  Track  resource  utilization  against  resource  estimations. 

The  Resource  Factor 

Criteria  for  the  resource  factor  considers  the  training,  experience,  and  morale  of  the  application 
staff,  budgei  constraints,  existence  of  adequate,  up-to-date  software  engineering  tools,  competing 
demands  placed  upon  the  application  staff,  the  adequacy  of  existing  hardware/software  config¬ 
urations,  and  the  availability  of  qualified  personnel.  The  final  score  for  the  resource  factor  can 
range  from  0  to  100.  The  following  interpretation  may  be  useful: 


78-100  Excellent  rating,  sufficient  resource  utilization 
51-75  Good  rating,  average  resource  efficiency  and  availability 
26-50  Mediocre  rating,  implement  improvements 
5-25  Poor  rating,  inadequate  resources 

Depending  on  the  severity  of  the  rating,  the  following  actions  may  be  taken  to  improve  the 
resource  factor.  This  list  does  not  represent  all  possible  appropriate  actions. 

•  Increase  programmer  training. 

•  Increase  morale,  reward  maintenance  efforts  over  development  efforts. 

•  Evaluate/install  new  software  engineering  tools. 

•  Increase  recruiting  efforts  to  obtain  qualified  programmers. 

•  Evaluate  existing  hardware/ software  configurations  against  the  existing  portfolio  of  infor¬ 
mation  systems  supported  by  the  organization. 

•  Minimize  the  number  of  languages  and  work  practices  used  for  implementation  in  order  to 
reduce  training  requirements. 


A  Glossary  of  Terms 


Acceptance  Review  A  review  of  a  software  product  by  developers  and  maintainers  to  deter¬ 
mine  if  the  product  satisfies  all  originally  specified  requirements. 

Acceptance  Test  Testing  led  by  the  client  or  QA  group  to  determine  whether  the  product 
satisfies  its  specifications  as  claimed  by  the  developer. [Sch90] 

Application  System  same  as  Information  System 

Availability  A  measure  of  the  degree  to  which  an  item  is  in  an  operable  and  committable  state 
at  the  start  of  a  mission  when  the  mission  is  called  for  at  a  random  point  in  time.[Dep82] 

Benchmark  Testing  Evaluation  of  the  system  performance  against  quantitative  requirements.[Sch90] 

Change  Request  Review  Board  An  authority  responsible  for  evaluating  and  approving  re¬ 
quests  for  changes  to  a  software  product. 

Cohesion  A  measure  of  the  degree  of  the  functional  relatedness  within  program  units.  [Som89] 

Complexity  A  characteristic  of  the  software  interface  which  influences  the  resources  another 
system  will  expend  or  commit  while  interfacing  with  the  software.  [CDS86] 

Configuration  Management  The  process  of  identifying  and  defining  the  configuration  items 
(hardware/software  units)  in  a  system,  controlling  the  release  and  change  of  these  items 
throughout  the  system  life  cycle,  recording  and  reporting  the  status  of  configuration  items 
and  change  requests,  and  verifying  the  completeness  and  correctness  of  configuration  items.[IEE83] 

Consistency  The  extent  to  which  uniform  design  techniques  and  notation  are  used.  [War87] 

Coupling  A  measure  of  the  strength  of  interconnections  (dependencies)  between  program  units. 
[Som89] 

Error  Human  action  that  results  in  software  containing  a  fault.  Examples  include  omission  or 
misinterpretation  of  user  requirements  in  a  software  specification,  incorrect  translation  or 
omission  of  a  requirement  in  the  design  specification.  [IEE83] 

Failure  A  departure  of  program  operation  from  program  requirements.[IEE83] 

Failure  Rate  The  number  of  failures  of  an  item  per  measure-of-life  uuit.[Dep82) 

Fault  A  manifestation  of  an  error  in  software.  A  fault,  if  encountered,  may  cause  a  failure. 
Synonymous  with  bug. 

Fourth  Generation  Language  (4GL)  A  computer  programming  language  that  provides  ab¬ 
stractions  of  data  and/or  procedural  specifications  and  is  usually  suited  for  a  particular 
application  domain. 

Integration  Testing  Verify  that  the  modules  of  the  system  combine  correctly  in  order  to  achieve 
a  product  that  meets  its  specifications.  [Sch90] 


IS  (Information  Systems)  Organization  An  organized  collection  of  procedures,  personnel, 
and  resources  dedicated  to  support  a  portfolio  of  information  systems. 

Lines  of  Code  Lines  of  source  code,  not  including  comments. 

Maintainability  The  probability  that  an  item  will  be  retained  in,  or  restored  to,  a  specified 
condition  within  a  given  period  if  prescribed  procedures  and  resources  are  used.[Dep82] 

Maintenance  All  actions  required  to  retain  an  item  in,  or  restore  it  to,  a  specified  condition. [Dep82] 

Maintenance  Audit  An  organized  review  of  the  maintenance  organization. 

Maintenance  Escort  Participation  of  the  software  maintainer  in  software  system  development. 

Man/Machine  Interface  The  software  that  supports  the  interaction  between  the  user  and  the 
system. 

Measure  A  high-level  unit  of  specification  which  characterizes,  evaluates,  or  predicts  various 
aspects  of  software  life  cycle  processes  and  products. 

Metric  A  measurable  indication  of  some  aspect  of  a  system.  [DeM82]  A  quantification  of  a 
specific  feature  of  the  software  life  cycle  process  or  software  product. 

Modularity  A  characteristic  of  software  such  that  it  is  well-structured,  highly  cohesive,  and 
minimally  coupled.  [War87] 

New  Systems  Development  The  development  of  a  system  which  has  never  been  fielded. 

Object  Oriented  Design  Designing  a  system  in  terms  of  abstract  data  types  where  the  objects 
are  instantiations  of  the  data  types  and  new  data  types  can  be  defines  as  extensions  of 
previously  defined  types. 

Regression  Testing  Testing  the  system  against  previous  test  cases  to  ensure  that  the  function¬ 
ality  of  the  system  has  not  been  compromised  by  recent  changes  to  the  system.  [Sch90] 

Reliability  The  probability  that  an  item  will  perform  its  intended  function  for  a  specified  interval 
under  stated  conditions.  [Dep82] 

Self-Descriptiveness  A  characteristic  of  software  that  enables  the  understanding  of  implemen¬ 
tation  of  software  functions.  [War87] 

Support  Staff  The  personnel  tasked  with  maintaining  an  information  system. 

Supportability  A  measure  of  the  adequacy  of  products,  resources,  and  procedures  to  facili¬ 
tate  the  support  activities  of  modifying  and  installing  software,  establishing  an  operational 
software  baseline,  and  meeting  user  requirements.  [PTH87] 

Testability  The  extent  to  which  software  facilitates  both  the  establishment  of  test  criteria  and 
the  evaluation  of  the  software  with  respect  to  those  criteria.  [IEE83] 

Throw-away  prototyping  Creating  a  prototype  as  part  of  system  design  and  then  "throwing 
away”  the  prototype  and  implementing  the  system  "from  scratch”  not  using  any  of  the 
source  code  from  the  prototype. 
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Top-down  design  Designing  the  system  by  recursively  breaking  the  system  down  into  smaller 
components. 

Unit  Testing  Testing  of  individual  portions  of  the  system. 


B  List  of  Acronyms 

AIRMICS  U.S.  Army  Institute  for  Research  in  Management  Information,  Communications, 
and  Computer  Science 

AMC  Army  Materiel  Command 

CCB  Change  Control  Board 

COE  Army  Corps  of  Engineers 

FORSCOM  Forces  Command 

HSC  Army  Health  Services  Command 

IS  Information  System 

ISC  Army  Information  Systems  Command 

LOC  Lines  of  Code 


C  Organizational  Questionnaire 


This  appendix  contains  a  5  page  questionnaire  for  gathering  information  about  the  support  organi¬ 
zation  in  order  to  calculate  the  supportability  measure.  The  questionnaire  should  be  photocopied 
and  distributed  to  selected  respondents. 


SSQAM  --  Supportability  Measure 
Organization  Questionnaire 

*★★★★★★★★**★**★*****★***★*★★*■*************★**★★★****★********★★*★★★★★★★★★***** 
QUESTIONNAIRE  NUMBER  _ 

1.  Which  of  the  following  organizational  techniques  are  established 
by  the  IS  organization  for  application  system  maintenance? 

Indicate  if  these  techniques  are  adequate. 

I 

I  Exist  (yes  or  no) 

I  _ 

I  I 

I  I  Adequate  (yes  or  no) 

I  I  _ 

I  1  ! 

I  I  I  Maintenance  escort  (participation  of  maintainer 

j _  | _ |  in  system  development.) 

Ill 

I  I  I  Acceptance  review  (in  transferring  software 

I _ I _ I  from  development  to  maintenance.) 

I  I  I 

I _ I _ I  Change  request  review  board  (i.e.  CCB) 

I  I  I 

I _ I _ I  Formal  retest  procedure  (in  implementing  changes) 

I  I  I 

I  I  I  Scheduled  maintenance  (changes  batched  and 

I _ I _ I  implemented  according  to  predetermined  schedule.) 

I  I  I 

I _ I _ |  Quality  assurance 

2.  Do  you  have  a  set  of  standards  to  follow  when  performing  the 
following  actions?  Are  these  standards  adequate? 


Exist  (yes  or  no) 

I 

I  Adequate  (yes  or  no) 

I  _ 

I  I 

_ I _ I  Developing  or  modifying  requirements  documentation 

I  I 

_ I _ I  Developing  or  modifying  design  documentation 

'll 

_ I _ I  Developing  or  modifying  the  man/machine  interface 

I  I 

_ I _ I  Developing  or  modifying  system  documentation 

I  I 

_ I _ I  Developing  or  modifying  source  code 

i  I 

_ I  _ I  Conducting  the  software  unit  tests 

I  I 

_ I _ I  Conducting  the  software  integration  tests 

I  I 

_ I _ I  Conducting  the  software  acceptance  tests 
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SSQAM  —  Supportability  Measure 
Organization  Questionnaire 


3.  Are  there  formalized  plans  or  procedures  for  conducting  the 
following  activities?  (Check  all  those  which  apply.) 

_  Monitoring  of  support  staff  performance 

_  Tracking  of  resource  utilization 

_  Coordination  of  tasks  within  the  support  staff 

_  Estimating  resources  (time,  personnel...)  necessary 

to  implement  changes  to  software 

_  Periodic  maintenance  audit 

_  Handling  user  change  requests 

_  Designing  new  systems  to  replace  existing  systems 

_  Monitoring  planned  maintenance  activities 

_  Configuration  management 

_  Creating  system  test  data 

_  Training  the  support  staff 

_  Training  the  system  user3 

4 .  Which  of  the  following  work  methods  are  established  by  the  IS 
organization  for  application  system  development  and  maintenance? 
(Check  all  those  which  apply.) 

_  Throw-away  prototyping 

_  Object  Oriented  Design  Methodology 

_  Top-down  design 

_  Regression  testing 

_  Benchmark  testing 

5.  What  is  the  current  total  number  of  (full-time  equivalent) 
application  systems  analysts  and  programmer  employees 

_  total  application  staff 
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SSQAM  —  Supportability  Measure 
Organization  Questionnaire 

***************r***«r***********************************W 


6.  What  is  the  length  of  service  (in  the  IS  Organization) 
distribution  of  the  current  application  staff?  Indicate  a 
percentage  for  each  category.  (Percentages  should  total 

to  100%) 

_  0-1  years 

_  1-3  years 

_  3-6  years 

_  6-10  years 

_  more  than  10  years 

7.  What  is  the  distribution  of  (immediate)  prior  job  experience  of 
the  current  appxxcacion  scarf?  Indicate  a  percentage  for  each 
category.  (Percentages  should  total  to  100%) 

_  position  in  other  IS  organization  within  parent  organization. 

_ _  other  position  within  parent  organization. 

_  position  in  other  IS  organization,  not  in  parent  organization. 

_  other  position,  not  in  parent  organization. 

_  no  prior  position  (student) . 


8.  What  is  the  distribution  of  educational  backgrounds  (highest  degrees 
obtained)  of  the  application  staff?  Indicate  a  percentage  for  each 
category.  (Percentages  should  total  to  100%) 

_  Graduate  college  degree 

_  Bachelors  college  degree 

_  Two-year  college  degree 

_  High  school  diploma  or  less 


SSQAM  —  Supportability  Measure 
Organization  Questionnaire 

ic^ic^iticititiriricicir^^ieirit^icic^icicic'k-k'kit-k-k'k-k-k-kiK'itir’k-k'k-k-k'k'k-kir'k'k-k'k'kic'k'k-kic'k'kic’k’k-k'k’k'k-k-k'kic’k'k'k’k’kif-kir 


9.  Overall,  in  your  judgement,  to  what  extent  are  (or  have  been)  the 
problems  in  maintaining  the  "irrent  installed  application 
portfolio?  (Check  the  appropriate  category.) 

No  Problem  At  All  | 


Somewhat  Minor  Problem 


Minor  Problem 


Somewhat  Major  Problem 


Major  Problem  | 


a.  Turnover  of  maintenance  personnel 


b.  Quality  of  application  system 
documentation 


c.  Changes  made  to  application 
system  hardware  and  software 


d.  User  demand  for  enhancements  and 
extensions  to  application  system 


e.  Skills  of  maintenance  programming 
personnel 


f.  Quality  of  original  programming 
of  application  system 


g.  Number  of  maintenance  programming 
personnel  available 


h.  Competing  demands  for  maintenance 
programming  personnel  time 


i.  Inadequate  hardware/ software 

configurations  in  IS  Organization 


j.  Ability  to  recruit  qualified 
personnel 


k.  Lack  of  user  understanding  of 
application  system 


1.  Storage  requirements  of 
application  system  programs 


m.  Inadequate  software  tools 

(debuggers,  analyzers,  etc..) 


n.  Motivation  of  maintenance  III) 

programming  personnel  I  I  I  I 
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SSQAM  --  Supportability  Measure 
Organization  Questionnaire 


No  Problem  At  All 


Somewhat  Minor  Problem 


Minor  Problem 


Somewhat  Major  Problem 


Major  Problem  1  1  1  1  i 

o.  Forecasting  of  maintenance  pro¬ 
gramming  personnel  requirements 

! 

! 

1 

1 

i 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

i 

i 

i 

i 

i 

i 

i 

i 

p .  Maintenance  programming 
productivity 

1 

t 

1 

1 

1 

1 

1 

1 

1 

1 

I 

1 

i 

i 

i 

i 

i 

q.  Competing  demands  between  new 
systems  development  and 
maintenance 

1 

i 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

i 

i 

i 

i 

i 

! 

z.  Turnover  in  user  organization 

1 

1 

1 

1 

1 

1 

i 

i 

1 

s.  Unrealistic  user  expectations 

i 

1 

1 

1 

1 

1 

i 

1 

1 

t .  Adherence  to  programming  standards 
in  maintenance 

I 

I 

1 

1 

1 

1 

1 

1 

1 

1 

1 

I 

i 

i 

i 

1 

1 

u.  Management  support  of  application 
system 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

i 

i 

i 

1 

1 

1 

v.  Quality  of  application  system 
design 

! 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

i 

i 

i 

i 

1 

1 

1 

w.  Budgetary  pressures 

! 

i 

1 

I 

1 

1 

1 

1 

i 

i 

1 

1 

x.  Meeting  scheduled  commitments 

1 

1 

1 

1 

1 

1 

i 

i 

i 

i 

1 

1 

y.  Inadequate  training  of  user 
personnel 

1 

1 

1 

1 

1 

1 

! 

1 

1 

i 

i 

i 

i 

i 

i 

1 

1 

1 
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D  System  Questionnaire 


This  appendix  contains  a  7  page  questionnaire  for  gathering  information  about  the  information 
system.  To  compute  the  supportability  measure  only  the  starred  (*)  questions  (questions  1- 
10,14,15,19,26a-j,26o)  need  to  be  completed.  If  you  are  computing  both  the  supportability  and 
operational  readiness  measures,  the  entire  system  questionnaire  needs  to  be  completed.  The 
questionnaire  should  be  photocopied  and  distributed  to  selected  respondents. 


Software  Supportability  Qualitative  Assessment  Methodology 

System  Questionnaire 


QUESTIONNAIRE  NUMBER:  _ 

Name  of  Information  System:  _ _ 

Software  and  Documentation  Information 

*1.  What  is  the  size  of  the  system  source  code,  in  lines  of  code  (LOC) ? 
_  lines  of  code 

*2.  What  language(s)  is  the  software  written  in? 


*3.  How  many  modules  (units  that  perform  single  functions  or  sets  of 
functions)  does  the  software  product  contain? 

_  number  of  modules 

*4.  What  is  the  age  (measured  from  date  of  original  installation) 
of  the  software  product? 

_  age  of  system  (in  years) 

*5.  How  long  has  your  organization  supported  this  software  product? 
_  length  of  support  (in  years) 

*6.  What  are  the  TOTAL  number  of  changes  that  have  been  made  to  this  product 
(software  and  associated  documentation)  during  the  time  you  have 
supported  it?  Include  both  Software  Change  Packages  and  Emergency 
Change  Packages. 

_  total  number  of  changes 

*7.  Does  the  software  contain  any  code  that  aids  in  debugging  the  software? 

_  yes 

no 
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Software  Supportability  Qualitative  Assessment  Methodology 

System  Questionnaire 


*8.  Is  there  any  documentation  explaining  the  overall  function  of  the 
software? 

_  yes 

no 


*9.  Is  there  documentation  for  each  module  explaining  the  module's  function? 

_ _  ye  3 

no 


*10.  Are  there  any  user's  manuals  explaining  the  use  of  this  software? 

_  yes 

no 


Maintenance  Information 


11.  For  what  amount  of  time  (how  many  hours)  during  the  month,  if  any,  is 
the  software  system  down  and  cannot  be  used? 

_  (hours)  down  time 

12.  What  is  the  average  number  of  maintenance  requests  per  month  received 
for  this  system? 

(Notes:  If  a  change  proposal  contains  several  requests,  count  each 

request  separately. 

Count  ALL  requests,  even  those  that  no  actions  are  taken  on.) 
_  average  number  of  maintenance  requests  per  month 

13.  Approximately  how  many  of  the  above  maintenance  requests  (per  month) 
ultimately  result  in  some  change  being  made  to  the  software? 

_  percentage  of  requests  (per  month)  which  result  in  changes 

to  the  software 

*14.  Approximately  what  percentage  of  the  maintenance  requests  FOR  WHICH 
YOU  PERFORM  ACTIONS  ON  are 

_  Small-scale  (affect  a  few  lines  of  code  at  most)? 

_  Medium-scale  (affect  several  functions  or  modules)? 

_  Large-scale  (affect  all  or  a  large  portion  of  the  software)? 

100  %  TOTAL 
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Software  Supportability  Qualitative  Assessment  Methodology 

System  Questionnaire 

★  ★*★*★★★★*★*★*★**★★**★■***★★*★****★****★**★*★*  **★★*★*★★*★★★*★★★*★★*★*★★★★★★**★* 


*15.  Approximately  what  percentage  of  the  maintenance  requests  FOR  WHICH 

YOU  PERFORM  ACTIONS  ON  are 

_  EMERGENCY  (require  immediate  attention  and  must  be  completed  as 

soon  as  possible  to  ensure  the  correct  operation  of  the  software) 

_  URGENT  (require  urgent  attention  -  more  so  than  normal  requests  - 

and  must  be  completed  within  a  relatively  short  period  of  time) 

_  NORMAL  (require  no  special  attention  and  can  be  completed  within 

the  usual  framework  of  support  procedures) 


100  %  TOTAL 


16.  What  percentage  of  ALL  maintenance  requests  you  receive... 

_  Are  for  corrections  to  faulty  software  components? 

_  Are  for  changes  (other  than  corrections)  or  enhancements  to  the 

software? 


100  %  TOTAL 


17.  What  percentage  (0-100%)  of  EMERGENCY  and  URGENT  requests  are  for 
corrections  to  faulty  software  components? 

_  percentage  of  EMERGENCY  and  URGENT  requests  that  are  corrections 


18.  ON  THE  AVERAGE,  what  percentage  (0-100%)  of  all  requests  require  more  time 
to  complete  than  is  originally  scheduled? 

_  percentage  of  all  requests  completed  behind  schedule 


*19.  What  percentage  of  time  spent  maintaining  the  software  is  devoted  to 
testing  it? 

_  (%)  time  spent  on  testing 


Software  Supportability  Qualitative  Assessment  Methodology 

System  Questionnaire 

****************************************************************************** 


User  Information 


20.  ON  THE  AVERAGE,  how  often  do  you  communicate  (either  formally  or 

informally)  with  a  TYPICAL  user  organization  using  this  information 
system?  Mark  the  one  appropriate  response  below. 

_  Several  times  a  day 

_  Once  or  twice  a  day 

_  At  least  weekly,  but  not  daily 

_  At  least  monthly,  but  not  weekly 

_  At  least  once  per  year,  but  not  monthly 

_  Less  than  once  per  year 


Current  Circumstances 


21.  How  many  people  in  your  support  organization  presently  maintain  this 
software  either  on  a  part-time  or  full-time  basis? 

(Indicate  the  number  in  each  category.) 

_  full-time  (number) 

_  part-time  (number) 


22.  AT  PRESENT  (NOT  on  the  average),  how  many  changes  of  all  types 

(including  corrections  and  enhancements)  are  there  to  be  implemented? 

_  number  of  changes  to  be  implemented 


23.  Of  the  above  changes  to  be  implemented,  what  percentage  (0-100%)  of 
these  changes  are  EMERGENCY  changes?  If  there  are  no  changes, 
answer  0%. 

_  percentage  of  current  changes  that  are  EMERGENCY 


24.  Of  the  changes  (from  #2)  to  be  implemented,  what  percentage  (0-100%) 
of  these  changes  are  for  CORRECTIONS  to  faulty  software  components? 
If  there  are  no  changes,  answer  0%. 

_  percentage  of  current  changes  that  are  CORRECTIONS 
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Software  Supportability  Qualitative  Assessment  Methodology 

System  Questionnaire 


25.  Based  on  the  following  scale,  how  you  you  rate  the  estimated  effort 
needed  to  complete  changes  to  the  software  product  over  the  next 
month : 

0  -  Much  more  effort  than  average 

1  -  Somewhat  more  effort  than  average 

2  -  Average  effort 

3  ■  Less  than  average  effort 

4  -  Much  less  than  average  effort 

5  -  No  effort  at  all  (no  changes  to  implement) 

_  answer  (0-5) 


Problem  Information 


*26.  Overall,  in  your  judgment,  to  what  extent  are  (or  have  been)  the 
following  problems  in  maintaining  this  information  system? 

(Check  the  appropriate  category.) 

No  Problem  At  All  | 

_  I 

Somewhat  Minor  Problem  |  | 

_  I  I 

Minor  Problem  |  I  I 

_  I  I  I 

Somewhat  Major  Problem  I  I  I  I 


Major  Problem  ( 


*a.  Not  enough  people  to  support  this! 
system.  ! 

1 

1  1  1  1  1 

1  I  1  1  1 

1  1  1  1  1 

*b.  People  supporting  this  system  are  I 
not  trained  adequately.  1 

1 

1  1  1  1  1 

1  1  1  1  1 

III!! 

*c.  System  is  overly  large,  making  | 

support  difficult.  I 

1 

1  1  1  1  1 

till! 

1  1  1  1  1 

*d.  System  is  overly  complex,  making  | 
support  difficult.  ! 

1 

!  1  1  1  1 

1  1  I  1  1 

I  1  1  1  1 

*e.  System  is  not  well-structured  I 

(written  in  "spaghetti  code")  .  i 

1 

1  1  1  1  1 

1  I  1  1  1 

I  I  1  1  I 

*f.  Lack  of  system  modularization  | 

makes  changes  difficult  to  I 

implement .  I 

1 

1  1  1  1  1 

1  1  1  1  1 

1  1  1  1  1 

1  1  1  1  1 
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Software  Supportability  Qualitative  Assessment  Methodology 

System  Questionnaire 


26  (cont'd) 


No  Problem  At  All 


Somewhat  Minor  Problem 


Minor  Problem 


Somewhat  Major  Problem 
Major  Problem  | 


g.  System  is  old  and  needs  to  be 
replaced. 

*h.  System  documentation  is 
incomplete  or  confusing. 

*i.  System  documentation  is  out-of- 
date  . 

*j.  Not  enough  time  is  spent  on 

testing  after  changes  are  made. 

k.  Software  repair  schedules  are 
hard  to  meet . 

l.  Overall,  there  are  more  change 
requests  submitted  for  this 
system  than  can  be  handled. 

m.  There  are  too  many  change 
requests  resulting  from  software 
bugs  (vs.  enhancement  requests). 

n.  There  are  too  many  emergency 
change  requests. 

*o .  User  requirements  for  this 
system  change  frequently. 
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Software  Supportability  Qualitative  Assessment  Methodology 

System  Questionnaire 


27.  Overall,  from  your  perspective,  to  what  extent  are  (or  have  been)  the 
problems  as  they  impact  on  the  ability  to  maintain  this  information 
system?  (Check  the  appropriate  category.) 


No  Problem  At  All 


Somewhat  Minor  Problem 


Minor  Problem 


Somewhat  Major  Problem 


Major  Problem  | 


a.  Skills  of  maintenance  programming 
personnel 

b.  Number  of  maintenance  programming 
personnel  available 

c.  Inadequate  hardware/software 
configurations  in  IS  Organization 

d.  Motivation  of  maintenance 
programming  personnel 

e.  Maintenance  programming 
productivity 

f.  Competing  demands  between  new 
systems  development  and 
maintenance 

g.  Budgetary  pressures 

h.  Meeting  scheduled  commitments 
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E  Scoring  Directions 

This  appendix  contains  the  following  four  items: 


1.  A  one-page  worksheet  for  recording  scores  from  the  organizational  questionnaire. 

2.  Nine  pages  of  directions  for  scoring  the  organizational  questionnaire. 

3.  A  one-page  worksheet  for  recording  scores  from  the  system  questionnaire. 

4.  Ten  pages  of  directions  for  scoring  the  system  questionnaire. 


SSQAM  —  Supportability  Measure 
Organizational  Questionnaire  Worksheet 


I*****************************************  ************** ******** 

QUESTIONNAIRE  NUMBER  _ 


Question 

System 

Process 

Resource 

Range 

1. 

1  1  1  1  II 

1  1  1  1  1  1  1  1 

0-12 

2. 

1  1  1  1  1  1 

1  1  1  1  1  1  1  1 

0-16 

3. 

1  1  1  1  1  1 

1  1  1  1  1  1  1  1 

0-12 

4. 

MINI 

1  1  1  1  1  1  1  1 

0-5 

5. 

1  1  1  1  1  1 

1  1  1  1  1  1  1 

1  1  1  1  1  1  1  1 

6. 

1  1  1  1  1  1 

1  1  1  1  1  1  1 

2-10 

7. 

1  1  1  1  1  1 

1  1  II  1  1  1 

2-10 

8. 

1  1  1  1  1  1 

1  !  1  1  1  1  1 

1-10 

9a. 

1  1  1  1  1  1 

1  1  1  II  1  1 

0-5 

b. 

1  I  1  1  1  1  1 

1  1  1  1  1  1  1  1 

0-5 

c . 

1  1  If  1  1  1 

1 1 1 1 1 II 1 

0-5 

d. 

1  1  1  1  1  1 

1  1  1  1  1  1  1 

0-5 

e . 

1  II  1  1  1 

1  1  1  1  1  1  1 

0-5 

f . 

1  1  1  1  1  1  1 

1  1  1  1  1  1  1  1 

0-5 

g- 

1  1  1  1  1  1 

1 1 1  1  1  II 

0-5 

h. 

1  1  1  1  1  1 

1  1  1  1  1  1  1 

0-5 

i . 

1  1  1  1  1  1 

1  1  1  1  1  1  1 

0-5 

j. 

1  1  1  1  1  1 

1  1  1  1  1  1  1 

0-5 

k. 

1  1  1  1  1  1 

1  1  1  1  1  1  1  1 

0-5 

1. 

1  1  1  1  1  1 

1  1  1  1  1  1  1 

0-5 

m. 

1  1  1  1  1  1 

1  1  1  1  1  1  1 

0-5 

n . 

1  1  1  1  1  1 

1  1  1  1  1  1  1  1 

0-5 

o . 

1  1  1  1  1  1 

1 II 1 1 1 1 1 

0-5 

P- 

1  1  1  1  1  1 

1  1  1  1  1  1  1 

0-5 

q  • 

1  II  1  1  1 

1  1  1  1  II  1  1 

0-5 

r . 

1  1  1  1  II 

1  1  1  1  1  1  1 

0-5 

s . 

1  1  1  1  1  1 

1  1  1  1  1  1  1  1 

0-5 

t . 

1  1  1  1  1  1 

1  1  1  1  1  1  1  1 

0-5 

u . 

1  1  1  1  1  1 

1  1  1  1  1  1  1  1 

0-5 

v . 

1  1  II  1  1  1 

1  II  1  II  1  1 

0-5 

w . 

1  1  1  1  1  1 

1  1  1  1  1  1  1 

0-5 

X  . 

1  1  1  1  1  1 

1  1  1  1  1  1  1  1 

0-5 

y . 

1  1  1  1  1  1 

1  1  1  1  1  1  1  1 

0-5 

TOTALS 

1  1  1  1  1  1  1 

1  1  1  1  1  1  1  1 

0-20 

1  1  1  1  1  1 

1  1  1  1  1  1  1  1 

0-90 

1  1  1  I  II 

1  1  1  1  1  1  1 

5-90 

SSQAM  —  Supportability  Measure 
Organization  Questionnaire  -  Scoring  Directions 

★★★a************************************************************************** 


QUESTIONNAIRE  NUMBER 


1.  Which  of  the  following  organizational  techniques  are  established 
by  the  IS  organization  for  application  system  maintenance? 
Indicate  if  these  techniques  are  adequate. 


Exist  (yes  or  no) 

I 

|  Adequate  (yes  or  no) 

I  _ 

I  I 

I  I  Maintenance  escort  (participation  of  maintainer 

_  I _ I  in  system  development.) 

I  I 

|  |  Acceptance  review  (in  transferring  software 

_ | _ |  from  development  to  maintenance.) 

I  I 

_ | _ |  Change  request  review  board  (i.e.  CCB) 

I  I 

_ | _ I  Formal  retest  procedure  (in  implementing  changes) 

I  I 

|  1  Scheduled  maintenance  (changes  batched  and 

_ | _ I  implemented  according  to  predetermined  schedule.) 

I  I 

_ | _ I  Quality  assurance 


SCORING 


For  each  affirmative  answer,  score  1  point. 
Maximum  of  12  points  possible. 


SCORE  - 
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SSQAM  —  Supportability  Measure 
Organization  Questionnaire  -  Scoring  Directions 

a***************************************************************************** 

2.  Do  you  have  a  set  of  standards  to  follow  when  performing  the 
following  actions?  Are  these  standards  adequate? 


Exist  (yes  or  no) 

I 

I  Adequate  (yes  or  no) 

I  _ 

I  I 

_ j _ I  Developing  or  modifying  requirements  documentation 

II 

_ I _ I  Developing  or  modifying  design  documentation 

II 

_ I _ I  Developing  or  modifying  the  man/machine  interface 

II 

_ I _ I  Developing  or  modifying  system  documentation 

I  I 

_ I _ I  Developing  or  modifying  source  code 

II 

_ I _ I  Conducting  the  software  unit  tests 

I  I 

_ I _ I  Conducting  the  software  integration  tests 

I  I 

_ I _ I  Conducting  the  software  acceptance  tests 


SCORING 

For  each  affirmative  answer,  score  1  point. 
Maximum  of  16  points  possible. 

SCORE  -  _ 
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SSQAM  —  Supportability  Measure 
Organization  Questionnaire  -  Scoring  Directions 

****************************************************************************** 

3.  Are  there  formalized  plans  or  procedures  for  conducting  the 
following  activities?  (Check  all  those  which  apply.) 

_  Monitoring  of  support  staff  performance 

_  Tracking  of  resource  utilization 

_  Coordination  of  tasks  within  the  support  staff 

_  Estimating  resources  (time,  personnel...)  necessary 

to  implement  changes  to  software 

_  Periodic  maintenance  audit 

_  Handling  user  change  requests 

_  Designing  new  systems  to  replace  existing  systems 

_  Monitoring  planned  maintenance  activities 

_  Configuration  management 

_  Creating  system  test  data 

_  Training  the  support  staff 

_  Training  the  system  users 


SCORING 

For  each  check,  score  1  point. 
Maximum  of  12  points  possible. 

SCORE  - 
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****************************************************************************** 

SSQAM  —  Supportability  Measure 
Organization  Questionnaire  -  Scoring  Directions 

****************************************************************************** 


4.  Which  of  the  following  work  methods  are  established  by  the  IS 

organization  for  application  system  development  and  maintenance? 
(Check  all  those  which  apply.) 

_  Throw-away  prototyping 

_  Object  Oriented  Design  Methodology 

_  Top-down  design 

_  Regression  testing 

_  Benchmark  testing 


SCORING 


For  each  check,  score  1  point. 
Maximum  of  5  points  possible. 


SCORE  - 


5.  What  is  the  current  total  number  of  (full-time  equivalent) 
application  systems  analysts  and  programmer  employees 

_  total  application  staff 


SCORING 


No  scoring  for  this  question. 
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SSQAM  —  Supportability  Measure 
Organization  Questionnaire  -  Scoring  Directions 


6.  What  is  the  length  of  service  (in  the  IS  Organization) 

distribution  of  the  current  application  staff?  Indicate  a 
percentage  for  each  category.  (Percentages  should  total 
to  100%) 

WEIGHT 


0-1  years  1 
1-3  years  2 
3-6  years  3 
6-10  years  4 
more  than  10  years  5 


SCORING 


For  each  percentage,  multiply  by  the  corresponding 
weight.  Sum  the  products  and  then  divide  the  sum 
by  50.  Maximum  score  is  10  points,  minimum  score 
is  2  points. 

Example:  For  these  percentages,  the  calculations  are 

as  follows: 


25% 

0-1 

years 

30% 

1-3 

years 

20% 

3-6 

years 

15% 

6-10 

years 

10% 

more 

than  10  years 

(25  *  1)  +  (30  *  2)  +  (20  *  3)  +  (15  *  4)  +  (10  *  5) 

50 


SCORE 


SSQAM  —  Supportability  Measure 
Organization  Questionnaire  -  Scoring  Directions 


7.  What  is  the  distribution  of  (immediate)  prior  job  experience  of 
the  current  application  staff?  Indicate  a  percentage  for  each 
category.  (Percentages  should  total  to  100%) 

WEIGHT 


position  in  other  IS  organization  5 

within  parent  organization. 

other  position  within  parent  organization.  3 

position  in  other  IS  organization,  not  in  3 

parent  organization. 

other  position,  not  in  parent  organization.  1 

no  prior  position  (student) .  1 


SCORING 


For  each  percentage,  multiply  by  the  corresponding 
weight.  Sum  the  products  and  then  divide  the  sum 
by  50.  Maximum  score  is  10  points,  minimum  score 
is  2  points. 

Example:  For  these  percentages,  the  calculations  are 

as  follows: 

25%  position  in  other  IS  organization  within  parent.. 

30%  other  position  within  parent  organization 

20%  position  in  other  IS  organization,  not  in  .... 

15%  other  position,  not  in  parent  organization 
10%  no  prior  position  (student) 


(25  *  5)  +  (30  *  3)  +  (20  *  3)  +  (15  *  1)  +  (10  *  1) 


50 


SCORE  - 
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★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★it 

8.  What  is  the  distribution  of  educational  backgrounds  (highest  degrees 
obtained)  of  the  application  staff?  Indicate  a  percentage  for  each 
category.  (Percentages  should  total  to  100%) 


WEIGHT 


Graduate  college  degree  10 
Bachelors  college  degree  8 
Two-year  college  degree  5 
High  school  diploma  or  less  1 


SCORING 


For  each  percentage,  multiply  by  the  corresponding 
weight.  Sum  the  products  and  then  divide  the  sum 
by  100.  Maximum  score  is  10  points,  minimum  score 
is  1  point. 

Example:  For  these  percentages,  the  calculations  are 

as  follows: 

15%  graduate  college  degree 
55%  bachelors  college  degree 
25%  two-year  college  degree 
5%  high  school  diploma  or  less 


(15  *  10)  +  (55  *  8)  +  (25  *  5)  +  (5  *  1) 


100 


SCORE 


****************************************************************************** 

SSQAM  —  Supportability  Measure 
Organization  Questionnaire  -  Scoring  Directions 

****************************************************************************** 

9.  Overall,  in  your  judgement,  to  what  extent  are  (or  have  been)  the 
problems  in  maintaining  the  current  installed  application 
portfolio?  (Check  the  appropriate  category.) 


No 

Problem  At  All 

1 

1 

Somewhat 

Minor 

Problem 

1 

1 

1 

1 

Minor 

Problem  | 

I 

1 

1 

1 

1 

Somewhat  Major 

Problem  | 

1 

1 

1 

1 

1 

1 

1 

Major  Problem  | 

1 

1 

1 

1 

SCORE 

a. 

Turnover  of  maintenance  personnel 

! 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

b. 

Quality  of  application  system 
documentation 

1 

1 

1 

1 

1 

1 

1 

1 

I 

1 

1 

1 

1 

1 

1 

1 

1 

1 

c . 

Changes  made  to  application 
system  hardware  and  software 

( 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

I 

1 

1 

d. 

User  demand  for  enhancements  and 
extensions  to  application  system 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

e. 

Skills  of  maintenance  programming 
personnel 

1 

I 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

I 

1 

f . 

Quality  of  original  programming 
of  application  system 

1 

1 

1 

1 

1 

I 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

g- 

Number  of  maintenance  programming 
personnel  available 

1 

1 

1 

1 

1 

1 

1 

1 

1 

! 

1 

I 

1 

1 

! 

1 

1 

1 

h. 

Competing  demands  for  maintenance 
programming  personnel  time 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

i . 

Inadequate  hardware/software 
configurations  in  IS  Organization 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

j. 

Ability  to  recruit  qualified 
personnel 

( 

1 

t 

1 

1 

I 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

k . 

Lack  of  user  understanding  of 
application  system 

1 

1 

I 

"  1 

1 

1 

I 

1 

1 

1 

1 

1 

I 

1 

1 

1 

1 

1 

1. 

Storage  requirements  of 
application  system  programs 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

i 

1 

1 

m. 

Inadequate  software  tools 
(debuggers,  analyzers,  etc..) 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

n . 

Motivation  of  maintenance 
programming  personnel 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 
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a***************************************************************************** 


No  Problem  At  All 


Somewhat  Minor  Problem 


Minor  Problem 


Somewhat  Major  Problem 


Major  Problem  | 


SCORE 


o.  Forecasting  of  maintenance  pro¬ 
gramming  personnel  requirements 

p.  Maintenance  programming 
productivity 

q.  Competing  demands  between  new 
systems  development  and 
maintenance 

z.  Turnover  in  user  organization 

s.  Unrealistic  user  expectations 

t .  Adherence  to  programming  standards 
in  maintenance 

u.  Management  support  of  application 
system 

v.  Quality  of  application  system 
design 

w.  Budgetary  pressures 

x.  Meeting  scheduled  commitments 

y.  Inadequate  training  of  user 
personnel 


SCORING 


For  each  lettered  item,  score 

5  points  for  "No  problem  at  all” 

4  points  for  "Somewhat  minor  problem" 

3  points  for  "Minor  problem" 

1  points  for  "Somewhat  major  problem" 

0  point  for  "Major  problem" 
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Software  Supportability  Qualitative  Assessment  Methodology 
System  Questionnaire  Worksheet  (Supportability) 


NAME  OF  SYSTEM  _ 

QUESTIONNAIRE  NUMBER 


Question 


System  Process  Resource  Range 


1.  I  I  I  I  I  I  I  I  II  I  I  I  I  I  0-5 

2.  I  I  II  I  I  I  I  I  I  I  I  I  I  I  0-5 

3.  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  0-5 

4.  I  I  I  I  I  I  I  II  ill  I  II  0-5 


5.  I  II  I  I  I  II  I  I  I  II  I  I  I  I  I  I  I  I 

6.  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  0-5 

7.  I  I  I  I  II  I  I  I  II  I  I  I  I  0,2 

8.  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  0,2 


9.  I  I  I  I  I  I  I  I  I  I  I  I  II  I  0,2 

10.  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  0,2 

14.  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  1-4 

15.  I  I  I  I  I  I  I  I  I  ||  I  I  I  I  1-4 

19.  I  I  I  I  III  I  I  I  I  I  I  I  I  0-4 


26a.  I  I  I  I  I  I  I  I  I  I  I  I  I  _  0-5 

b.  I  I  I  I  I  I  I  I  I  I  I  I  I  _  0-5 

c.  _  I  I  I  I  I  I  I  II  I  I  I  I  I  I  0-5 

d.  _  I  I  II  I  I  I  I  I  I  I  I  I  I  I  0-5 

e.  II  I  I  I  II  I  I  III  I  I  I  0-5 


f  .  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  0-5 

g.  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  0-5 

h.  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  0-5 

i.  I  III  I  I  I  I  II  I  I  I  I  I  0-5 

j.  I  I  I  I  I  I  _  I  I  I  I  I  I  I  I  0-5 

o.  I  I  I  I  I  I  _  I  I  I  I  I  I  I  I  0-5 


TOTALS  _  I  I  I  I  II  I  I  II  I  I  I  I  I  2-80 

I  I  I  I  I  I  _  I  I  II  I  I  II  0-10 

II  I  I  I  I  I  I  I  II  I  I  0-10 


Software  Supportability  Qualitative  Assessment  Methodology 
System  Questionnaire  -  Scoring  Directions  (Supportability) 

****************rt****************************************************** 

QUESTIONNAIRE  NUMBER  _ 

Name  of  Information  System:  _ _ 


1.  What  is  the  size  of  the  system  source  code,  in  lines  of  code  (LOC) ? 

lines  of  code 


SCORING 


Calculate  the  score  utilizing  the  following  scale: 


System  Size 


At  least  But  less  than  SCORE 


0  10,000  lines  of  code  5 

10,000  50,000  "  4 

50,000  100,000  "  3 

100,000  500,000  "  2 

500,000  1,000,000  "  1 

1,000,000  ”  0 


SCORE  - 
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Software  Supportability  Qualitative  Assessment  Methodology 
System  Questionnaire  -  Scoring  Directions  (Supportability) 


2.  What  language (s)  is  the  software  written  in? 


SCORING 


Scoring  should  be  based  upon  the  following  scale: 


Number  of  languages  SCORE 


1  4 

2  3 

3  2 

4  1 

Greater  than  4  0 


Add  one  additional  point  to  the  score  if  at  least  half  of 
the  languages  are  high-level,  4th  generation  languages 
or  later. 


Examples  of  allowable  languages 


Language 


COBOL 

Assembly 

C 

Ada 

DBASE  III 
SQL 


Allowed? 


No 

No 

No 

Yes 

Yes 

Yes 


SCORE  - 
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Software  Supportability  Qualitative  Assessment  Methodology 
System  Questionnaire  -  Scoring  Directions  (Supportability) 


3.  How  many  modules  (units  that  perform  single  functions  or  sets  of 
functions)  does  the  software  product  contain? 

number  of  modules 


SCORING 


To  calculate  a  score  for  this  question,  you  need  the  answers 
for  both  this  question  and  question  number  one  (system  size 
in  lines  of  code) . 

Calculate  the  average  module  size,  in  lines  of  code,  by  dividing 
the  answer  in  number  one  by  the  answer  to  this  question.  Then 
assign  a  score  according  to  the  following  scale: 

Average  Module  Size 


Least 

But  Less  Than 

SCORE 

0 

500  lines  of  code 

5 

500 

1,000 

4 

1,000 

2,000 

3 

2,000 

3,000 

2 

3,000 

5,000 

1 

5,000 

11 

0 

Example:  If  the  information  system  contains  200,000  lines 

of  code  and  it  contains  300  modules,  then  the 
average  module  size  is: 

200,000  /  160  -  1,250  lines  of  code. 

Thus,  we  would  assign  a  score  of  3. 


SCORE  - 


*  ★  it  ★ 
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Software  Supportability  Qualitative  Assessment  Methodology 
System  Questionnaire  -  Scoring  Directions  (Supportability) 


4.  What  is  the  age  (measured  from  date  of  original  installation) 
of  the  software  product? 

_  age  of  system  (in  years) 


SCORING 


Compute  the  score  using  the  following  scale: 
System  Age 


At  Least  But  Less  Than 


0 

1 

3 

6 

8 

10 


1 

3 

6 

8 

10 


year 

years 


SCORE 

5 

4 

3 

2 

1 

0 


SCORE  - 


5.  How  long  has  your  organization  supported  this  software  product? 
_  length  of  support  (in  years) 


SCORING 


No  scoring  for  this  question. 
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Software  Supportabllity  Qualitative  Assessment  Methodology 
System  Questionnaire  -  Scoring  Directions  (Supportability) 


6.  What  are  the  TOTAL  number  of  changes  that  have  been  made  to  this  product 
(software  and  associated  documentation)  during  the  time  you  have 
supnorted  it?  Include  both  Software  Change  Packages  and  Emergency 
Change  Packages. 

_ _  total  number  of  changes 


SCORING 


To  compute  the  score  for  this  question,  you  need  the  answers 
for  both  this  question  and  question  number  5  (length  of  support) . 

Calculate  the  average  number  of  changes  per  year  by  dividing 
the  answer  to  this  question  by  the  answer  in  number  5.  Then 
assign  a  score  according  to  the  following  scale: 

Average  Number  of 
Changes  Per  Year 

At  Least  But  Less  Than  SCORE 


0 

5 

changes  per  year 

5 

5 

10 

II 

4 

10 

50 

II 

3 

50 

100 

11 

2 

100 

500 

It 

1 

500 

It 

0 

Example:  If  the  information  system  has  been  supported  for 

5  years,  and  a  total  of  175  changes  have  been 
implemented  to  this  system,  then  the  average 
number  of  changes  is: 

175  /  5  -  35  changes  per  year. 

Thus,  we  would  assign  a  score  of  3. 


SCORE  - 
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Software  Supportability  Qualitative  Assessment  Methodology 
System  Questionnaire  -  Scoring  Directions  (Supportability) 


7.  Does  the  software  contain  any  code  that  aids  in  debugging  the  software? 

_  yes  SCORE  _ 

_  no 


8.  Is  there  any  documentation  explaining  the  overall  function  of  the 
software? 

_  yes  SCORE  _ 

no 


9.  Is  there  documentation  for  each  module  explaining  the  module's  function? 

_  yes  SCORE  _ 

_  no 

10.  Are  there  any  user's  manuals  explaining  the  use  of  this  software? 

_  yes  SCORE  _ 

no 


SCORING 


For  each  of  questions  7  through  10,  assign  a  score  of  2  points 
for  each  "yes"  answer  and  a  score  of  0  points  for  each  "no" 
answer . 
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Software  Supportability  Qualitative  Assessment  Methodology 
System  Questionnaire  -  Scoring  Directions  (Supportability) 


14.  Approximately  what  percentage  of  the  maintenance  requests  FOR  WHICH 
YOU  PERFORM  ACTIONS  ON  are 

_  Small-scale  (affect  a  few  lines  of  code  at  most)? 


Medium-scale  (affect  several  functions  or  modules)? 


Large-scale  (affect  all  or  a  large  portion  of  the  software)? 


100  %  TOTAL 


SCORING 


For  each  percentage,  multiply  by  the  corresponding  weight 
as  shown  in  the  following  weight  table: 


Type  of  Action  WEIGHT 


Small-Scale  4 
Medium-Scale  3 
Large-Scale  1 


Sum  the  resulting  products,  divide  the  result  by  100, 

and  round  to  the  nearest  integer.  Maximum  score  is  4  points, 

minimum  score  is  1  point . 

Example:  If  the  percentage  for  the  various  types  of 

maintenance  requests  are  as  follows: 

45%  Small-scale  requests 
45%  Medium-scale  requests 
10%  Large-scale  requests 


The  calculation  is: 

(45  *  4)  +  (45  *  3)  +  (10  *  1) 
_  -  3.25, 

100 

which  rounds  to  3.  Thus,  the  score  is  3. 


SCORE  = 


Page 


****************************************************************************** 

Software  Supportability  Qualitative  Assessment  Methodology 
System  Questionnaire  -  Scoring  Directions  (Supportability) 

****************************************************************************** 


15.  Approximately  what  percentage  of  the  maintenance  requests  FOR  WHICH 
YOU  PERFORM  ACTIONS  ON  are 

_  EMERGENCY  (require  immediate  attention  and  must  be  completed  as 

soon  as  possible  to  ensure  the  correct  operation  of  the  software) 


URGENT  (require  urgent  attention  —  more  so  than  normal  requests 
and  must  be  completed  within  a  relatively  short  period  of  time) 

NORMAL  (require  no  special  attention  and  can  be  completed  within 
the  usual  framework  of  support  procedures) 


100  %  TOTAL 


SCORING 


For  each  percentage,  multiply  by  the  corresponding  weight 
as  shown  in  the  following  weight  table: 


Type  of  Request  WEIGHT 


Emergency  1 
Urgent  2 
Normal  4 


Sum  the  resulting  products,  divide  the  result  by  100, 

and  round  to  the  nearest  integer.  Maximum  score  is  4  points, 

minimum  score  is  1  point. 


Example:  If  the  percentage  for  the  various  types  of 

maintenance  requests  are  as  follows: 

10%  Emergency 
10%  Urgent 
80%  Normal 


The  calculation  is: 

(10  *  1)  +  (10  *  2)  +  (80  *  4) 

_  -  3.50, 


100 


which  rounds  to  4.  Thus,  the  score  is  4. 


SCORE  - 
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Software  Supportability  Qualitative  Assessment  Methodology 
System  Questionnaire  -  Scoring  Directions  (Supportability) 

****************************************************************************** 


19.  What  percentage  of  time  spent  maintaining  the  software  is  devoted  to 
testing  it? 

_  (%)  time  spent  on  testing 


SCORING 


To  calculate  the  score  for  this  question,  utilize  the  following 
scale: 


Percentage  Test  Time 


At  Least 

But  less  Than 

SCORE 

0% 

10% 

0 

10% 

20% 

1 

20% 

30% 

2 

30% 

40% 

3 

40% 

4 

SCORE  = 
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Software  Supportability  Qualitative  Assessment  Methodology 
System  Questionnaire  -  Scoring  Directions  (Supportability) 


***********«**«**************„****to*M„„„„MHOMHMMHMtttMtttH 
26.  Overall,  in  your  judgment,  to  what  extent  are  (or  have  been)  the 
following  problems  in  maintaining  this  information  system? 


No  Problem  At  All 


Somewhat  Minor  Problem 


Minor  Problem 


Somewhat  Major  Problem 
Major  Problem  ( 


a.  Not  enough  people  to  support  this 
system. 

b.  People  supporting  this  system  are 
not  trained  adequately. 

c.  System  is  overly  large,  making 
support  difficult. 

d.  System  is  overly  complex,  making 
support  difficult. 

e.  System  is  not  well-structured 
(written  in  "spaghetti  code”) . 

f .  Lack  of  system  modularization 
makes  changes  difficult  to 
implement . 

g.  System  is  old  and  needs  to  be 
replaced. 

h.  System  documentation  is 
incomplete  or  confusing. 

i.  System  documentation  is  out-of- 
date  . 

j .  Not  enough  time  is  spent  on 
testing  after  changes  are  made. 

o.  User  requirements  for  this 
system  change  frequently. 


SCORE 


SCORING 


For  each  lettered  item,  score 

5  points  for  "No  problem  at  all" 

4  points  for  "Somewhat  minor  problem" 
3  points  for  "Minor  problem" 

1  point  for  "Somewhat  major  problem" 
0  points  for  "Major  problem" 
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F  Supportability  Worksheet  -  Final  Results 

This  appendix  contains  a  one- page  worksheet  for  calculating  the  final  supportability  measure. 


**************** 


Software  Supportability  Qualitative  Assessment  Methodology 
Supportability  Worksheet 

******************************************^^^^^^^^^^^#^^ 


NAME  OF  SYSTEM 


System  Process  Resource 

AVERAGE  Organizational 
Questionnaire  Scores 


AVERAGE  System 

Questionnaire  Scores 


TOTAL  Factor  Scores 


\ 

\ 

\ 

\ 

\ 

\ 

\ 

\ 

\ 


/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 


\ 


/ 


\  / 


+ 


ADD  Factor  Scores 


DIVIDE  by  3 

3 


FINAL  SCORE 


i 
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